A storage system typically comprises one or more storage devices into which information may be entered, and from which information may be obtained, as desired. The storage system includes a storage operating system that functionally organizes the system by, inter alia, invoking storage operations in support of a storage service implemented by the system. The storage system may be implemented in accordance with a variety of storage architectures including, but not limited to, a network-attached storage (NAS) environment, a storage area network (SAN) and a disk assembly directly attached to a client or host computer. The storage devices are typically disk drives organized as a disk array, wherein the term “disk” commonly describes a self-contained rotating magnetic media storage device. Optical media are also envisioned. The disk or disk array in this context apply to a hard disk drive (HDD) or direct access storage device (DASD).
Storage of information on the disk array is preferably implemented as one or more storage “volumes”, defining an overall logical arrangement of disk space. The disks of a volume are typically organized as one or more groups, wherein each group may be operated as a Redundant Array of Independent (or Inexpensive) Disks (RAID). Most RAID implementations enhance the reliability/integrity of data storage through the redundant writing of data “stripes” across a given number of physical disks in the RAID group, and the appropriate storing of redundant information (parity) with respect to the striped data. The physical disks of each RAID group may include disks configured to store striped data (ie., data disks) and disks configured to store parity for the data (ie., parity disks). The parity may thereafter be retrieved to enable recovery of data lost when a disk fails, or for other purposes.
The storage operating system of the storage system may implement a high-level module, such as a file system, to logically organize the information stored on the disks as a hierarchical structure of directories, files and blocks. For example, each “on-disk” file may be implemented as set of data structures, i.e., disk blocks, configured to store information, such as the actual data for the file. These data blocks are organized within a volume block number (VBN) space that is maintained by the file system. The file system organizes the data blocks within the VBN space as a “logical volume”; each logical volume may be, although is not necessarily, associated with its own file system. The file system typically consists of a contiguous range of VBNs, from 0 to n, for a file system of size n−1 blocks.
A known type of file system is a write-anywhere file system that does not overwrite data on disks. If a data block is retrieved (read) from disk into a memory of the storage system and “dirtied” (i.e., updated or modified) with new data, the data block is thereafter stored (written) to a new location on disk to optimize write performance. A write-anywhere file system may initially assume an optimal layout such that the data is substantially contiguously arranged on disks. The optimal disk layout reduces fragmentation, resulting in efficient access operations, particularly for sequential read operations, directed to the disks.
The storage operating system may further implement a storage module, such as a RAID system, that manages the storage and retrieval of the information to and from the disks in accordance with input/output (I/O) operations. The RAID system is also responsible for parity operations in the storage system. Note that the file system only “sees” the data disks within its VBN space; the parity disks are “hidden” from the file system and, thus, are only visible to the RAID system. The RAID system typically organizes the RAID groups into one large “physical” disk (i.e., a physical volume), such that the disk blocks are concatenated across all disks of all RAID groups. The logical volume maintained by the file system is then “disposed over” the physical volume maintained by the RAID system.
The storage system may be configured to operate according to a client/server model of information delivery to thereby allow many clients to access the directories, files and blocks stored on the system. In this model, the client may comprise an application, such as a database application, executing on a computer that “connects” to the storage system over a computer network, such as a point-to-point link, shared local area network, wide area network or virtual private network implemented over a public network, such as the Internet. Each client may request the services of the file system by issuing file system protocol messages (in the form of packets) to the storage system over the network. By supporting a plurality of file system protocols, such as the conventional Common Internet File System (CIFS) and the Network File System (NFS) protocols, the utility of the storage system is enhanced.
When accessing a block of a file in response to servicing a client request, the file system specifies a VBN that is translated at the file system/RAID system boundary into a disk block number (DBN) location on a particular disk (disk, DBN) within a RAID group of the physical volume. Each block in the VBN space and in the DBN space is typically fixed, e.g., 4 k bytes (KB), in size; accordingly, there is typically a one-to-one mapping between the information stored on the disks in the DBN space and the information organized by the file system in the VBN space. The (disk, DBN) location specified by the RAID system is further translated by a disk driver system of the storage operating system into a sector (or similar granularity) on the specified disk.
The requested block is then retrieved from disk and stored in a buffer cache of the memory as part of a buffer tree of the file. The buffer tree is an internal representation of blocks for a file stored in the buffer cache and maintained by the file system. Broadly stated, the buffer tree has an Mode at the root (top-level) of the file. An inode is a data structure used to store information, such as metadata, about a file, whereas the data blocks are structures used to store the actual data for the file. The information contained in an inode may include, e.g., ownership of the file, access permission for the file, size of the file, file type and references to locations on disk of the data blocks for the file. The references to the locations of the file data are provided by pointers, which may further reference indirect blocks that, in turn, reference the data blocks, depending for example upon the quantity of data in the file. Each pointer may be embodied as a VBN to facilitate efficiency among the file system and the RAID system when accessing the data on disks.
The file system, such as the write-anywhere file system, maintains information about the configuration of the underlying physical disks (e.g., the number of blocks in each disk) in the storage system. The RAID system provides the disk geometry information to the file system for use when creating and maintaining the VBN-to-disk, DBN mappings used to perform write allocation operations. The file system maintains block allocation data structures, such as an active map, a space map, a summary map and snapmaps. These mapping data structures describe which blocks are currently in use and which are available for use and are used by a write allocator of the file system as existing infrastructure for the logical volume.
Specifically, the snapmap denotes a bitmap file describing which blocks are used by a consistency point. The write-anywhere file system has the capability to generate a consistency point of its active file system. An “active file system” is a file system to which data can be both written and read or, more generally, an active store that responds to both read and write I/O operations. A consistency point is a space conservative, point-in-time read-only image of data accessible by name that provides a consistent image of that data (such as a storage system) at some previous time. More particularly, a consistency point is a point-in-time representation of a storage element, such as an active file system, file or database, stored on a storage device (e.g., on disk) or other persistent memory and having a name or other identifier that distinguishes it from other consistency points taken at other points in time. A consistency point can also include other information (metadata) about the file system at the particular point in time for which the image is taken.
The write-anywhere file system supports multiple consistency points that are generally created on a regular schedule, or at prescribed points in time or events. Each consistency point refers to a copy of the file system that diverges from the active file system over time as the active file system is modified. Each consistency point is a restorable version of the storage element (e.g., the active file system) created at a predetermined point in time and, as noted, is “read-only” accessible and “space-conservative.” Space conservative denotes that common parts of the storage element in multiple consistency points share the same file system blocks. Generally, only the differences among these various consistency points require extra storage blocks. The multiple consistency points of a storage element are not independent copies, each consuming disk space; therefore, creation of a consistency point on the file system is instantaneous, since no entity data needs to be copied. Read-only accessibility denotes that a consistency point cannot be modified because it is closely coupled to a single writable image in the active file system. The closely coupled association between a file in the active file system and the same file in a consistency point obviates the use of multiple “same” files.
The active map denotes a bitmap file describing which blocks are used by the active file system. As described prior, a consistency point may contain metadata describing the file system as it existed at the point in time that the image was taken. In particular, a consistency point captures the active map as it existed at the time of consistency point creation; this file is also known as the snapmap for the consistency point. Note then that a snapmap denotes a bitmap file describing which blocks are used by a consistency point. The summary map denotes a file that is an inclusive logical OR bitmap of all snapmaps. By examining the active and summary maps, the file system can determine whether a block is in use by either the active file system or any consistency point. The space map denotes a file including an array of numbers that describe the number of storage blocks used in a block allocation area. In other words, the space map is essentially a logical OR bitmap between the active and summary maps to provide a condensed version of available “free block” areas within the vbn space.
FIG. 1 is a schematic block diagram of an exemplary on-disk storage structure 100 of a logical volume of a storage system. As noted, a logical volume is typically associated with a file system and comprises data blocks organized within a VBN space. Each logical volume (hereinafter “volume”) has a file system information (fsinfo) block that is preferably stored at a fixed location within, e.g., a RAID group. Fsinfo block 105 is the root of the on-disk storage structure 100, illustratively at VBNs 1 and 2. When loading the volume, the storage operating system accesses those VBNs to acquire the fsinfo block 105.
The fsinfo block 105 includes a variety of metadata that describes the state of the file system; also included in the fsinfo block 105 is an inode for an inode file 110. Inodes of the write-anywhere file system are organized into the inode file 111. Like any other file, the inode of the inode file is the root of the buffer tree that describes the location of blocks of the file. As such, the inode of the inode file may directly reference (point to) data blocks 107 of the inode file 111 or may reference indirect blocks 106 of the inode file 111 that, in turn, reference data blocks of the inode file. In this example, the inode for the inode file 110 includes an exemplary buffer tree comprising a plurality of inode file indirect blocks 106 that, in turn, point to inode file data blocks 107. Within each data block of the inode file are inodes 112, each of which serves as the root of a file. Among the inodes of the inode file 110, there are inodes for special metadata files, such as an active map 115, a summary map 120, a space map 125, a root directory 140 and a metadata directory 145. All user files in the file system are organized under the root directory 140, while various metadata files associated with the file system are stored under the metadata directory 145.
The inode file may further include inodes that reference a plurality of consistency points 130, 135. These consistency point inodes are the root level inodes of consistency points of the active file system. Each volume has special reserved inode numbers within its VBN space; a plurality of those inode numbers (e.g., 31) is reserved for consistency points. When a consistency point is generated of the active file system, a copy of the inode for the inode file is generated (hereinafter the “consistency point root”) and assigned one of the reserved consistency point inode numbers. Thus, to access a consistency point at a particular point in time, the storage operating system accesses the appropriate consistency point root of the consistency point.
With system-wide static consistency points, the system is incapable of preventing one volume from consuming all or most of the resources leaving none for other volumes. When this occurs and the volume is unable to release buffers and non-volatile logs (NVLogs) in a timely manner, and operations for that volume experience latency due to lack of resources. Further, other volumes experience operational latency due to the behavior of this volume. Adding to the unpredictable nature of consistency points are the storage server internal operations, which may run during the consistency point. These internal workloads generate more work for the running consistency point, leading to unpredictable consistency point duration, which translates to inconsistent performance within the storage server. The present disclosure is directed to alleviating this limitation.
The present disclosure is susceptible of various modifications and alternative forms, and some representative embodiments have been shown by way of example in the drawings and will be described in detail herein. It should be understood, however, that the inventive aspects are not limited to the particular forms illustrated in the drawings. Rather, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the appended claims.